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Abstract 

The goals of this case study were to: (1) determine the efficiency and effectiveness of three survey 
methods — postal mail survey, web-based sun’ev, and random in-class administration survey — in assessing tobacco- 
related attitudes and behaviors among college students and (2) compare the response rate and procedures of these 
three methods. There was a large disparity in response rate between the three survey methods. In-class 
administration had the highest response rate (66%), followed by postal mail survey (23%) and web-based survey 
(10%). Based on the results, we recommend in-class survey administration with random class selection for 
conducting survey research on college campuses. This method appears to be efficient, productive and simple, and 
does not allow for the occurrence of many of the difficulties that commonly arise when other modes are used. 
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Introduction 

Survey research is often used to evaluate health 
education and health promotion projects, 
particularly among college students. The goals of 
this case study were (1) to determine the 
efficiency and effectiveness of three survey 
methods - postal mail survey, web-based survey, 
and random in-class administration survey - in 
assessing tobacco -related attitudes and behaviors 
among college students and (2) to compare the 
response rate and administrative procedures of 
these three methods. 

College -aged students are the youngest 
population able to legally purchase tobacco 
products and are, therefore, a major target for the 
tobacco industry. Many states have partnered 
with colleges and universities to reduce tobacco 
use among college students (1). The anti- 
tobacco programs and activities conducted on 
college campuses usually are evaluated for 
effectiveness, efficiency, and fidelity to program 
goals and objectives. A survey research was 
conducted as part of a comprehensive evaluation 
of the anti-tobacco program. 

The Florida Annual Tobacco Survey 
(FACTS) is used to assess tobacco -related 
attitudes and behaviors among college students 
in Florida. The Florida Department of Health, 
Division of Health Awareness and Tobacco, 
sponsors an annual project, Student Tobacco 
Reform Initiative- Knowledge for Eternity 
(SR1KE) aimed to reduce tobacco use and 
improve tobacco-free environment among 
college campuses in Florida. During the 2001- 
2002 school year, this survey was administered 
via three modes: postal mail survey, web-based 
survey, and random in-class administration 
survey. Each survey method was examined via 
literature review for advantages, disadvantages, 
general utility, and generalizability of data. 
Guidelines for various survey methods then were 
developed to maximize the response rate. 

Methods 

Instrument 

The questionnaire used in the study was 
developed based on the goals and objectives of 
the College Advisory Initiative (CA1), previous 
FACT questionnaires, feedback from the anti- 
tobacco program coordinators, and the college 
tobacco -use literature. This instrument 
consisted of 43 questions and 146 items 
(variables) that assessed demographic data, 


tobacco knowledge, attitude towards tobacco, 
college and university policies use, and personal 
tobacco use. The Institutional Review Board of 
the University of Florida approved the project. 

The instrument was reviewed by an 
expert panel for “clarity of instructions,” 
“reliability and utility,” “adequacy and 
appropriateness of questions and responses,” and 
“layout and attractiveness.” The reliability of 
the current FACTS survey was established with 
a pilot study using a test-retest reliability method 
with seventy-eight students enrolled in a 
Personal and Family Health Class. Correlation 
coefficient (r) ranged from .80 to 1.0 for 
demographic and background questions. Kuder 
and Richardson 20 (K-R 20) at .97 indicated a 
high level or reliability for knowledge questions. 
Both test-retest reliability and Chronbach’s alpha 
coefficient were used to establish the reliability 
of the attitudes questions. Test-retest reliability 
indices showed high significance level (p< .001) 
on each of the questions (r ranged from .53 to 
.70). 

Procedure and Sample Selection 

Eleven colleges and universities in the 
state university system participated in the study. 
Three hundred students were randomly selected 
from each school for participation. Each 
institution selected one of the three survey 
methods that best suited their institution’s need, 
based on level of technology, staff time, and 
other factors. The registrar from each school 
generated the random lists of students for 
schools that used mail surveys and web-based 
surveys. Each school was asked to send a letter 
or email to students to notify them of the 
upcoming study and to encourage participation. 
Mails surveys were sent to home rather than 
campus addresses. They also were asked to use 
follow-up measures such as phone calls, letters, 
or email to increase response rate. Guidelines 
for conducting postal mail survey are listed 
below. 

Guidelines for Conducting Mail Survey 
(FACTS) 

1. Obtain names, classifications, 
mailing addresses, and telephone 
numbers of 300 randomly selected 
students from your campus. 

2. Send a copy of the sample list to 
the evaluation team. 

3. Send a postcard to selected students 
one week before the survey. The 
post card should include 
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information about the STRIKE 
program and ask for students 
support to complete a 
questionnaire. The message should 
also include any incentives you 
plan to use to encourage 
completion of the survey. 

4. Receive a package of 300 printed 
survey forms from the evaluation 
team. 

5. Site coordinator and staff should 
attempt to complete the survey first 
to experience the time it takes to 
complete the survey. 

6. Mail a copy of the survey to each 
selected student with a self 
addressed and stamped return 
envelope. 

7. Collect and mail the completed 
surveys back to the evaluation 
team. 

8. A list of students who have not 
completed the survey will be 
provided to each site for follow up. 

Students who received the web-based 
surveys received an email and an individualized 
personal identification numbers that functioned 
as passwords to the online survey. Guidelines 
for conducting web-based survey are listed 
below. 


Guidelines for Conducting Web-Based Survey 
(FACTS) 

1. Obtain names, classifications, 
email addresses, street addresses, 
and telephone numbers of 300 
randomly selected students from 
your campus. 

2. Send a copy of the sample list to 
the evaluation team. 

3. Send an email or a postcard to 
selected students one week before 
the survey. The email should 
include information about STRIKE 
and ask for students’ support to 
complete an electronic survey, 
which will be sent by the 
evaluation team within a week. The 
email should also include any 
incentives you plan to use to 
encourage completion of the 
survey. The web address for the 


survey is 

http://www.hht). ufl.edu/FACTS . 

4. The introduction email should 
serve a purpose to verify the email 
address. Notify the evaluation team 
if there are any incorrect or 
undeliverable email addresses of 
selected students. 

5. All sites are encouraged to provide 
an incentive for completion the 
survey (e.g., STRIKE phone card 
or other STRIKE items). 

6. Site coordinator and staff should 
attempt to complete the survey first 
to experience the time it takes to 
complete the survey. 

7. The evaluation team will send out 
the survey with a code number via 
email one week after the advance 
introduction letter. The evaluation 
team will responsible for tracking 
the responses. 

8. A list of students who have not 
completed the survey will be 
provided to each site for follow up. 

The research team selected a random 
list of 30 courses from the electronic course 
listings of each school choosing the in-class 
paper survey method. Permission was obtained 
from instructors to use their classes. If an 
instructor refused, another course was randomly 
selected. Campus coordinators provided various 
incentives such a movie passes, phone cards, and 
coupons to complete the surveys. The response 
rate was assessed by using the number of usable 
returned surveys divided by the number of 
surveys distributed. The guidelines for random 
in-class survey are listed below. 

Guidelines for Random In-Class Survey 
(FACTS) 

1. A list of randomly selected classes 
will be provided to each site. 

2. Contact your registrar’s office to 
verify the courses selected. 

3. Select classes from the list until 
you reach 300 students. 

4. Ask your Vice-President or Dean to 
write an introductory letter to 
request for cooperation and support 
to class instructors selected for 
survey. 

5. Obtain instructor’ s permission to 
survey. If an instructor refuses to 
allow you to survey in the 
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classroom then ask for permission 
to distribute the survey only (before 
the class). Ask students to complete 
the survey at their own time and 
return the completed survey by 
campus mail or at a designated 
place in exchange for a gift. You 
can also ask students to bring back 
to class next time. 

6. Skip to next class on the list if you 
are not successful in getting the 
instructor’s permission to survey in 
the classroom or distribute the 
survey. 

7. Receive a package of surveys from 
the evaluation team. Each package 
will contain 300 survey forms to be 
distributed to students in each class. 

8. Site coordinator and staff should 
attempt to complete the survey first 
to experience the time it takes to 
complete the survey. (The survey 
should take only 15 minutes or less 
to complete.) 

9. You are encouraged to provide an 
incentive for completion of survey 
(e.g., STRIKE phone card or other 
STRIKE items). 

10. Collect the survey upon 
completion. Try to avoid collecting 
the survey on Friday. 

1 1 . Mail surveys back to the evaluation 
team. A specific instruction for 
returning the survey will be 
included in the survey package. 

Results 

The in-class administered surveys had 
the highest average response rate of the three 
methods (66%). In addition, the four schools 
that used this mode also had the highest response 
rates in the study. These schools did not use pre- 
notification or follow-up measures to increase 
response rate. Only one of these schools used an 
incentive. Mail surveys had the second highest 
response rate (23%). Two schools used this 
method and both used incentives. One school 
made two follo\wup calls to students and the 
other school sent a follow-up letter. Web-based 
surveys had the lowest response rate (10%). 
Five schools used this method but only three 
used pre -notification measures. All five schools 
offered an incentive to students who completed 
the survey. Follow-up methods differed for each 
school. One school provided an email follow- 


up, two schools made two email follow-ups, and 
one school made two phone follow-ups. One 
school did not provide a follow-up contact. 

Discussion 

In-Class Surveys 

This study showed a large disparity in 
response rate between the three survey methods. 
In-class survey administration had the highest 
response rate. This is a very popular survey 
method that traditionally yields high response 
rates (2-4). The classroom setting provides a 
captive audience and a quiet environment 
relatively free of distractions (2). In addition, 
this method is relatively inexpensive since it 
saves the cost of postal mailings and phone calls. 
Data can be processed almost immediately. The 
endorsement of the instructor also may provide 
legitimacy to the study in the eyes of students. 
Randomly selected classes capture all the above 
advantages plus it allows for generalization (2). 
Of course, the research question and adequate 
number of classes from which to sample will 
determine the appropriateness of this method. 

A major drawback of in-class survey 
administration is the difficulty of obtaining a 
truly random sample of individuals. While a 
random sample of individuals can be selected by 
postal mail or email surveys, classroom surveys 
randomly select classes, not individuals. This 
mode also relies upon the cooperation of 
instructors and assumes that the day of the 
survey represents normal class attendance. 
Additionally, students may be more likely to 
provide a higher rate of socially desirable 
responses (2). In comparison to web-based 
surveys, in-class surveys may require additional 
data entry time and labor costs. These problems 
can be minimized by having students record their 
responses on electronic scan sheets. 

Postal Mail Surveys 

For this study, there was a very low 
response rate with postal mail surveys (23%). 
Postal mail survey is the most commonly used 
survey method in research studies. It also is a 
method that produces low response rates (5). 
Advantages of the postal mail method include 
the ability to randomly select participants and a 
lack of technical barriers that may exist in the 
use of telephone or web-based survey 
administration. In addition, it is an excellent 
way for dealing with sensitive issues. 

Disadvantages include a slower rate of 
completion, the cost of printing additional 
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questionnaires, the postage cost of initial and 
follow-up mailings, and the potential for non- 
response bias (4). 

Postal surveys administered to groups 
with moderate levels of education, such as 
college students, have achieved response rates as 
high as 50 - 70% (6-8). It is not clear why the 
response rate for this study was so low. One 
possible explanation is that many students still 
use their parents’ home address for their mail, 
rather than their off-campus address, so they may 
not have received the questionnaire. In addition, 
completing a tobacco survey may not be 
considered a priority to college students. 
Multiple mailings may have increased the 
response rate. 

Web-Based/Online Surveys 

The web-based survey had a very low 
response rate (10%). Many studies have found a 
lower response rate for web-based surveys when 
compared to traditional paper-and-pencil survey 
and mail surveys (9,10). One study (9) reported 
a 19% response rate in a study of college 
students. Other researchers have reported 
response rates comparable to those with mail 
surveys (11,12). However, some researchers 
believe that there is insufficient empirical data to 
assert that web-based surveys can consistently 
obtain response rates as great as mail surveys 
(13,14). The use of incentives or promotional tie- 
ins has been recommended to increase response 
rates (9). 

Web-based surveys offer a faster 
dissemination, completion, and analysis rates 
than any other survey method (13,15-17). They 
also are relatively inexpensive to conduct, do not 
require trained interviewers, and eliminate the 
cost of postage (14,17,18). Web-based data 
collection also yields fewer socially desirable 
responses and fewer skipped items than other 
self-administered survey methods and can 
provide a relatively anonymous setting in which 
a participant may divulge personal information, 
yielding potentially more truthful responses (17). 

Major problems with web-based 
surveys usually include a poorly designed 
survey, technical problems in administering and 
completing the survey, defining the survey 
population, adequate coverage of the population, 
and selecting a scientifically valid sample 
(15,18). This issue is minimized if students 
complete the survey on their personal computers. 
Coverage and sampling issues can be minimized 
with college students since most have access to 
computers and the Internet. Other potential 


obstacles to web-based survey administration, 
specific to the college population, include lack of 
computer experience, not checking email 
regularly, computer systems which purge old 
email regularly, the lack of a visual reminder, the 
potential lack of privacy for students who 
complete the survey in a campus computer lab 
and even the potential for socially desirable 
responses (11,18). Many students also tend to 
have multiple email addresses and this may have 
contributed to the extremely low response rate in 
this study (11). The response rate for this 
method may have improved if all schools had 
made initial and follow-up contacts. Also, 
following-up with another method other than 
email may have helped (19). One school chose 
to follow-up with phone calls. 

Summary and Conclusions 

This study had several limitations. Due 
to the nature of the contract between the funding 
agency and the schools, each school was 
responsible for distributing the surveys, making 
initial and follow-up contacts with students, and 
providing incentives. Thus, there were added 
levels of bureaucracy that limited the control of 
the research team. These issues have proved to 
be especially crucial for the postal mail and web- 
based survey methods. Therefore, it can be 
concluded that the response rates, especially for 
the postal mail and web-based surveys, were 
probably affected by how the questionnaires 
were administered by each school. 

Based on this study, we recommend in- 
class survey administration with random class 
selection as a sound method for conducting 
survey research on college campuses. This 
method appears to be efficient, productive and 
simple, and does not allow for the occurrence of 
many of the difficulties and low response rates 
that commonly arise when other modes are used. 
As demonstrated by the high response rate 
obtained in this study, using the classroom 
distribution method was clearly the best choice. 

Regardless of the survey method used, 
research done on college campuses also should 
consider the best time in the semester to conduct 
the survey. If conducted too early in the 
semester, the class rolls may be incomplete due 
to adding and dropping of classes. We 
recommend randomly selecting individuals and 
classes after the fifth week of the semester. 
Researchers also should be familiar with the 
academic calendar of each school and plan 
around mid-term exams, final exams, winter 
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break, spring break, homecoming, or other 
events, which may interfere with students’ 
ability to give attention to the survey in a timely 
manner. 
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